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METHOD FOR DISPLAY TIME STAMPING DISCLOSURE OF INVENTION 

AND SYNCHRONIZATION OF MULTIPLE The above problems can be solved by implementing a 

VIDEO OBJECT PLANES common temporal resolution to be used for all the local time 

base. In order to cater for the wide range of temporal 

TECHNICAL FIELD 5 granularity the local time base is partitioned into two dif- 

This invention is useful in the coding of digital audio fe^^nt parts. The first part ^°"'^"g 

visual materials where there is a need for synchronizing tempo^l resolution which provide a shon ame b^e The 

, . ^ . , J J J* • «i ^u-^^Z second part contam a coarse granulanty temporal resolution 

more than one mdependently encoded audio visual objects ^ecunu pmi v^unumi ^ k 

- r . n i-i u ^^^1 which orovide a Ions time base. The snort time case is 

for presentation. It IS espeaally suitable when the temporal wm^ii piw iv*^ « ^ ... ^ 

lui picaciiiaiawi , , - i- oor«« 10 included at cach the vidco objcct planes to providc temporal 

samolinff of the audio visual matenal is not the same. ui^iu^w i ,u -j^^ «u;^^*« tu^ i^i,«rt 

^ ^ ^ reference to the instances of the video objects. The snort 

BACKGROUND ART ^ synchronized to the longtime base which 

is common to all the video object planes. It is used to 
In MPEG-1 and MPEG-2 the input video consists of synchronize all the different video object planes to a com- 
image frames sampled at regular time intervals. This repre- 15 ^on time base provided by the master clock, 
sents the finest temporal resolution of the input FIG. 1 shows According to a fist aspect of the present invention, a 
a video sequence with fixed frame rates where the image metiiod of encoding a local time base of an audio visual 
frames are sampled at regular intervals. In the coded repre- sequence in the compressed data comprises the steps of: 
sentation of the video sequence using the MPEG-1 and obtaining instances of the audio visual sequence by tem- 
MPEG-2 standards die display order of the decoded frames 20 ^ sampling; 

is indicated by the temporal reference. THis parameter determining ihe local time base of the said instances to be 
appears at the picture header of the bii^tream syntax. Tlie ^^^^^^^ i„to compressed data; 

O f''^ i,^' ^T^'^' ^. T'T AniL''^^^ encoding the said local time base in two parts, comprising 

decoded frame when exammed m the display order. ^ ^^J^ ^^^^^ ^ 

In the H263 standard it is possible to skip frames and thus -5 j^fg^nce spaced evenly at a specific interval on the 

. - decode a variable frame rate video sequence. However the j^^^ ^ increment relative to the said 

UJ sampling of frames is stiU fixed. Thus the temporal reference ^ ^^^^ reference; 

m method used in MPEG-1 and MPEG-2 is still appropnate ^^^^^ .^^^ compressed data 

and only need to be modified such that die mcrement is not ^^^^^^J^^ ^^^^ h^^e elapsed; and 

P in steps of 1 but m steps of plus the number of non- 30 P ^^^^^^ compressed 

: -7 transmitted pictures at the mput frame rate. J ^^^^ ^^^^ ^^^^^^^ 

Currently work is being doiie in die area of codmg video According to a second aspect of the invention, a method 

u as separate objects m muiuple video object planes. Tins encoding a local time base of an audio visual sequence in 
represents a new dmiension in the decoding sy^^^^^ 35 ^^e compressed data comprises the steps of: 

mzaaonof the respective video o^^^^^-^^^^^^^^ obtairSig instances of the audio visual sequence by tern- 

: these different video object planes may come from several uuvaiiiuig n 

sources and can have very different frame rates. Some of the PO^al samphng; r ,u ;«cro«o.c 

m objects may be rendered and have almost continuous tern- determmmg the local uine base of the said mstances to be 

H poral sampling rate. These video objects planes are com- encoded mto compressed data; 

".s^ bined into a composite image for display. Therefore some encoding the said instancesin one of two methods com- 
kind of synchronization is required for the composition. It is prising of a first mediod of compression without reference to 
possible that the display frame rate is different from any of any ftiture instances and a second method of compression 
the frame rates of the video object planes. HG. 2 shows an which referTuy die future recoristructed mstance; 
example of two video object planes having different and encoding the said local time base in two parts, comprising 
varying frame rates compared to one another. Even if a of a modulo time base that marks the occurrence of a set of 
common frame rate can be found between the two video time reference spaced evenly at a specific interval on die 
object plane, it is not automatic that tiiis frame rate will be local time base and a time base increment 
the same as the output frame rate of die compositor. encoding the time base increment for the instance corn- 
Hereafter we wiU refer to the problem in die video pressed using the first compression mediod as an absolute 
domain. However die same principles of this invention can value relative to die said evenly spaced time reference; 
be extended to the audio domain as well as combinations of encoding die time base increment for die instance com- 
tjjg pressed using the second compression mediod as a relative 
From die above it is clear diat die current state of die art value to die local time base of die previously compressed 
does not cater for die synchronization of video object planes. 55 instance using die said first mediod; 
The state of die art also does not provide a common temporal inserting die modulo time base mto die compressed data 
reference when die different video object planes have dif- whenever die specific interval have elapsed; and 
ferent frame rates which are not multiples of one anodier. inserting die time base increment within the compressed 

The first problem is how to provide a common local time data of die said instances of die audio visual sequence, 

base mechanism for each video object plane. This time base 60 According to a diird aspect of die invention, a mediod of 

will have to be able to provide very fine temporal granularity die first or die second aspect of die invention, wherein a 

and at die same time be able to cope widi possibility of very plurality of compressed bitstreams containing local time 

long intervals between two successive instances of die video base information encoded dierein are multiplexed, further 

object planes. comprises, for multiplexing, die steps of: 

The second problem is how to provide a mechanism for 65 synchronizing die local time base of die individual corn- 
synchronizing video object planes having different frame pressed bitstream to a common time base by encodmg and 

inserting die time base offset in the multiplexed bitstream; 



rates. 




6,07 

3 

examining each of the compressed bitstream for the next 
compressed instance to be placed into the multiplexed 
bitstream until all the compressed bitstreams have arrived at 
the modulo time base; 

inserting a common modulo time base into the multiplex 
bitstream and skipping the modulo time base of the com- 
pressed bitstreams; and 

repeating the last two steps until all compressed bit- 
streams are exhausted. 

According to a fourth aspect of the invention, a method of 
the first or die second aspect of the invention, wherein a 
plurality of compressed bitstreams containing local time 
base information encoded therein are multiplexed and 
de-multiplexed, further comprises, for de-multiplexing, the 
steps of: 

decoding the time base offset of the individual com- 
pressed bitstream; 

examining the multiplexed bitstream for the next com- 
pressed instance and placing the said instance into the 
appropriate compressed bitstream until a modulo time base 
is encountered in the multiplexed bitstream; 

inserting a modulo time base to each of the compressed 
bitstreams; and 

repeating the last two steps until the multiplexed bitstream 
is exhausted. 

According to a fifth aspect of the invention, a method of 
decoding a local time base of an audio visual sequence from 
the time base of the compressed data encoded, according to 
the first aspect of the invention, comprises the steps of: 

initializing the reference time base taking into account die 
time base offset; 

incrementing the reference time base by a specific interval 
for each modulo time base decoded; 

decoding the time base increment of the compressed 
instance; and 

determining the decoding time base of the said instance 
by adding the said decoded time base increment value to the 
reference time base. 

According to a sixth aspect of the invention, a method of 
decoding a local time base of an audio visual sequence from 
the time base of the compressecr'iLita encoded, according to 
the second aspect of the invention, comprises the steps of 

initializing the reference time base taking into accotmt of 
the time base offset; 

incrementing the reference time base by a specific interval 
for each modulo time base decoded; 

decoding the time base increment of the compressed 
instance; and 

determining the time base increment to be one of two 
types, absolute or relative, depending on the first or second 
compression method used in the encoding of the instance, 
respectively; 

determining the decoding time base of the said instance 
by adding the said decoded time base increment value to the 
reference time base if the time base is of the first type; and 

determining the decoding time base of the said instance 
by adding the said decoded time base increment value to the 
decoding time base of the previous instance encoded using 
die first compression method, if die time base increment is 
of die second type. 

BRIEF DESCRIPTION OF DRAWINGS 
The present invention wiU become more fully understood 
from die detailed description given below and die accom- 
panying diagrams wherein: 
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FIG. 1 illustrates the temporal sampling of the prior art 
where the frames of the video sequence is sampled at regular 
intervals. 

FIG. 2 illustrates the concept of video object planes and 
5 die relationship with one anodier. The sampling of die video 
object planes can be irregular and die sampling period can 
vary drastically. 

FIG. 3A iUustrates the current invention where die tem- 
poral reference of the video objects are indicated by the 
modulo time base and die VOP time increment. Illustration 
uses I and P VOP only. 

FIG. 3B illustrates the current invention where the tem- 
poral reference of the video objects are indicated by the 
modulo time base and die VOP time increment. Illustration 
uses I, P and B VOR 

FIG, 4 illustrates an example of die ambiguity diat can 
arise when die presentation order and the encoding order is 
different as a result of B-video object planes. 
20 FIG. 5 illustrates a solution to die ambiguity by employ- 
ing absolute and relative time bases. 

FIG. 6 illustrates the combining of two VOPs and their 
synchronization to the common time base by employing 
VOP time offsets. 

FIG. 7 illustrates the flow diagram for die encoding of the 
time base. 

FIG. 8 illustrates the flow diagram for die multiplexing of 
more than one video object planes. 
30 FIG. 9 iEustrates die flow diagram for the de-multiplexing 
of more than one video object planes. 

FIG. 10 illustrates the flow diagram for die recovery of the 
presentation time stamp. 
FIG. 11 illustrates the block diagram for die operations of 
35 the bitstream encoder for encoding of die time base. 

FIG. 12 illustrates the block diagram for die operations of 
die bitstream decoder for decoding of die time base. 
FIG. 13 illustrates the time chart for forming a bitstream 

40 

BEST MODE FOR CARRYING OUT THE 
INVENTION 

This invention operates by providing two forms of syn- 
45 chronization. The first is a short time base attached to the 
video object plane. Hereafter this time base will be referred 
to as die VOP time mcrement The VOP time increment is 
the timing mechanism for the video object plane with 
respect to the long time base which is attached to a group of 
50 video object planes to be decoded and composed together. 
The long time base is referred to as the modulo time base. 
The VOP time increment and die modulo time based will 
dien be used joindy to determine die actual time base to use 
for the composition of the video object planes into die final 
55 composited sequence for display. 

In order to facilitate editing of the bitstream and combin- 
ing different video objea planes from different sources into 
a new group of video object planes there is a need for a diird 
component to allow a fixed amount of offset between the 
60 local time base of the individual video object plane from the 
common time base. Hereafter diis offset will be referred to 
as die VOP time offset This prevents die different video 
object planes from having to synchronize at a granularity 
equivalent to die modulo time base interval. This component 
65 should be kept constant for each video object plane within 
the group of video object planes diat are multiplexed 
together. 
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First, the modulo time base is described. 
Modulo time base represent the coarse resolution of the 
local time base. It does not have a value like the VOP time 
increment. In fact it is more of a synchronization mechanism 
to synchronize the VOP time increment to the local time 
base of the video object plane. It is placed in the encoded 
bitstream as markers to indicate that the VOP time increment 
of die video object planes that follows are to be reset and that 
the reference to the local time base are to be incremented by 
one or more units of the modulo time base interval. In FIGS. 
3A» 3B, 4» 5 and 6, die modulo time base is shown as a series 
of zero or more "1" terminated by a **0" inserted into the 
bitstream header before die VOP time increment The num- 
ber of "1" inserted into the bitstream depends on the number 
of units of modulo time base that has elapsed since the last 
code I or P-video object plane. At the encoder and decoder 
the modulo time base counter is incremented by one for each 
"1" encountered. Modulo time based counter is finite in 
length and therefore in a practical system the modulo time 
base is reset to zero when the maximum value is exhausted. 
In a typical video sequence, the video object planes form a 

group of VOP. Therefore die modulo time base is usually 

reset at the start of this group of VOP. 

Next, the VOP time increment is described. 
The VOP time increment shall be in units that can support 

the shortest temporal sampling of the video objects. It can 

also be the native time base for the object which is rendered. 

It therefore represents the finest granularity of temporal 

resolution that is required or can be supported. 

The VOP time increment can dien be indicated by a finite 

length number greater or equal to die ratio of the global time 

base interval to die local time base resolution. FIG. 3A 

shows an example of the VOP time increment for I and 

P-video object planes and the reference to the modulo time 35 

base. An absolute time base is used The VOP time incre- 
ment is reset each time the modulo time base is encountered. 

FIG. 3B shows another example using I, P and B-video 

object planes. The operation is the same with the exception 

diat the modulo time base is repeated in die B-video object 40 current value of the local time base. In step 3 die local time 
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type. Therefore the solution is to code die VOP time 
increment of die B-video object planes as a relative value to 
the preceding I or P-video object planes and that the modulo 
time base applies only to the I and P-video object planes and 
not die B video object planes. This solution is depicted in 
HG. 5. 

Next, VOP time offset is described. 
Besides the above the modulo time base are shared 
between aU video object planes. This means that the syn- 
chronization of the different video objects will have a 
granularity equivalent to the modulo time base interval. This 
is unacceptable especially in die case where video object 
planes firom different groups are combined to form a new 
group of video object planes. FIG. 6 shows an example of 
two different video object planes that were encoded with two 
different local time base that are offset from one another. 
Thus when the video object planes are multiplexed, die 
synchronization of the video objects planes are also offset 
A finer granularity is achieved by allowing each individual 
video object plane to have a VOP time offset. This means 
that only this value is changed when die video object planes 
are manipulated and multiplexed. There is no need to change 
die VOP tune increment and yet it is possible to multiplex 
die different video object planes without coarse graniilarity 
timing differentials. FIG, 6 illustrates die use of this time 
base offsets. 

The preferred embodiment of the current invention 
involves a method of encoding the time base for each of the 
individual video object plane bitstream, a mediod of multi- 
plexing different video object planes into a common time 
base, a method of de-multiplexing die multiplexed bitstream 
into the components and a method of recovering die time 
base from the component bitstream. 
Encoding die tijne base is described. 
The flow diagram of the embodiment of the time base 
encoding is shown in FIG. 7. In the encoder die local time 
base is first initialized to the local start time in step 1. The 
process is passed to step 2 where die encoder determines the 



planes. If die modulo time base is not repeated in die 
B-video object plane then ambiguities arise due to die 
difference in decoding and presentation order. This is 
described in detail below. 

Since the VOP time increment corresponds to the presen- 
tation time base, a potential problem will arise when the 
encoding order is different from presentation order. This 
occurs with B-video object planes. Similar to MPEG-1 and 
MPEG-2 B-pictures, B-video object planes are encoded 



base obtained is compared to the previously encoded 
modulo time base to see if die interval has exceeded the 
modulo time base interval. If the interval has-beeiife.3S5eded 
the control is passed to step 4 where the required number of 
45 modulo time base is inserted into the bitstreariL If the 
interval has not been exceeded then no special handling is 
necessary. The processing then proceeds to step 5 where the 
VOP time increment is inserted into the bitstream. The 
object is then encoded in step 6 and the inserted into tiie 



after dieir reference I and P-video object planes even diough 50 bitstream. The encoder dien checks to determine if diere are 



more objects to be encoded in step 7. If there are more 
objects to be encoded die process resumes at step 2 where 
die local time base is obtained. If diere are no "more objects 
to be encoded then the process terminates. 

The following formula are used to determine the absolute 
and relative VOP time increments for I/P-video object 
planes and B-video object planes, respectively. 



diey preceded die reference video objects in die presentation 
order. Since die VOP time increment is finite and is relative 
to the modulo time base, the VOP time increment is reset 
when the modulo time base is encountered. However, the 
order of encoding for die B-video object planes have been 55 
delayed. FIG. 4 show the ambiguity that can arise. It is not 
possible to determine when the VOP time increment should 
be reset In fact when given a sequence of encoded events as 
in FIG. 4A, it is not possible to know which of tiie timing 
situations of FIGS. 4B, 4C and 4D it is trying to represent 60 
This problem comes about because of die use of one modulo 
time base shared among aU die different types of video 
object coupled with the difference between die encoding and 
presentation order. There is nothing diat can be done to die 
encoding order since the reference information is required 65 where 

by die B-video object planes. It is also not desirable to have tcrsn is ^® encoder time base marked by die ndi encoded 
independent modulo time base for die different prediction modulo time base. 



'<7r8«=n>^£7ra/+'c7Tw>(«=0. I* 2. 3, ... ) 
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t^sf is the pre-determined modulo time base interval, 
^cTBo ^ encoder time base start time, 
t^yj-/ is the absolute VOP time increment for the I or P- video 
object planes, 

^ETBi/p is the encoder time base at the start of die encoding 

of the I or P video object planes, 
t^vTY is the relative VOP time increment for die B-video 

object planes, and 
^ETBB is the encoder time base at the start of the encoding of 

the B video object plane. 

Next, multiplexing more than one video object planes is 
described. 

When more than one video object planes are multiplexed 
together, the multiplexer exanunes die multiple video object 
plane bitstreams to determine the order of multiplexing as 
well as die synchronization. The operations involved is 
depicted in FIG. 8. The VOP time offsets for each of the 
video object planes to be multiplexed are insened into the 
bitstream in step 11. All die bitstreams of the video object 
planes to be multiplexed are then examined in step 12 to 
determine if ail of the planes are at (heir respective modulo 
time base. If they are then the processing is passes to step 13 
where a common modulo time base is inserted into the 
multiplexed bitstream. The processing then proceeds to step 
14 ,where the next encoded video object is inserted into the 
multiplexed bitstream. In step 15 die bitstreams of the video 
object planes to be multiplexed are again examined to see if 
there are more video objects to be multiplexed. If diere is, 
dien the control is passed to step 12 again. Otherwise die 
process ends. 

De-multiplexing the bitstream containing more than one 
video object planes is described. 

The de-multiplexing of the bitstream containing multiple 
video object planes is depicted in FIG. 9. The process begins 
in step 21 where the VOP time offsets are decoded and 
passed to the decoders for synchronization. Then the mul- 
tiplexed bitstream is examined in step 22 to see if the 
modulo time base is found. If the modulo time base is found 
then the process is passed to step 23 where the modulo time 
base is inserted into all the video object bitstreams. The 



The following formula are used to determine the presen- 
tation time stamp of the video objects: 



fGTBn=f^^cr8r^^CTB0 («=0' K 2. 3. . 
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where 

^OTBn is decoding time base marked by the nth decoded 

modulo time base, 
^cTBi is pre-determined modulo time base interval, 
^CTBQ is the decoding time base start time, 
^DTBUP is die decoding time base at the start of the decoding 

of die I or P video object planes, 
t^vr/ is the decoded absolute VOP time increment for the I 

or P-video object planes, 
^DTBB is the decoding time base at the start of the decoding 

of die B video object plane, and 
t^vr/ is the decoded relative VOP time increment for the 

B-video object planes. 

Implementation of the Bitstream Encoder is described. 
FIG. 11 shows the block diagram for an implementation 
of the bitstream encoder for encoding the modulo time base 
and the VOP time increment. For the purpose of this 
illustration the example in FIG. 3B is used. Since 
bi-directional prediction is used, the encoding order is 
different from the presentation order shown in FIG. 3B. The 
encoding order begins widi die I-VOP followed by the 
P-VOP before die B-VOP This is illustrated in die three 
paragraphs below. 

The process begins in the initializer, step 41 where the 
bitstteam encoder starts by initializing die local time base 
register to the initial value of the time code. Ibe same time 
code value is encoded into the bitstream. At die start of the 
encoding of die next I-VOP the time code comparator, step 
42, compares die presentation time of die I-VOP against the 
local time base register. The result is passed to the modulo 



processing dien continues widi step 24 where die next video 40 time base encoder, step 43. The modulo time base encoder 
object is examined and inserted into die appropriate video will insert into die bitstream die required number of '' V 
object bitstream. Finally the multiplexed bitstream is exam 



ined again to see if there is any more video object to 
de-multiplex. If there is, dien the process is passed to step 22 
again. Otherwise the process ends. 
Recovering die time base is described. 
The embodiment of die time base recovery is shown in 
FIG. 10. In recovering the local time base, the process 
begins in step 31 where die local time based is initialized 
taking into account of die VOP time offset decoded by die 
de-multiplexer. Then the process is passed to step 32 which 
checks the bitstream to determine if the modulo time base is 
decoded. If the modulo time base is decoded then the 
process is passed to step 33 where the local tiime base is 
incremented by die modulo time base increment. The pro- 
cess is then passed to step 37. If the modulo time base is not 
decoded then the process is passed to step 34 where die 
video object is examine to determine if it is a B-video object 
or not If it is then die process is passed to step 35 where die 
B-video object decoding time base is calculated based on 
equation (6). The process is then passed to step 37. If the 
result of step 34 is not a B-video object dien die process is 
passed to step 36 where the decoding time base is calculated 
based on equation (5). The process then proceeds to step 37. 
In step 37. the bitstream is examine to see if there is more 
video objects to decode. If there is then the process is passed 
to step 32 again. Odierwise die processing ends. 
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equivalent to the number of modulo time base increment that 
has ela^s^ed. This is tiien followed by the symbol "O" to 
signal die end of the modulo time base code. The local time 
base register is updated to the current modido time base. The 
processing then proceed to the VOP time base increment 
encoder, step 44, where the remainder of die presentation 
time code of the 1-VOP is encoded. 

The process then repeats with die next encoded yideo 
object plane which is die P-VOP. The time code comparator, 
step 42. compares the presentation time of the P-VOP 
against the local time base register. The result is passed to 
the modulo time base encoder, step 43. The modtdo time 
base encoder will insert die reqtiired number of "1" eqiuva- 
lent to the number of modulo time base increment that has 
elapsed. This is dien followed by the symbol "O" to signal 
die end of the modulo time base code. The B-VOP time base 
register is set to the value of the local time base register and 
the local time base register is updated to die current modulo 
time base. The processing then proceeds to the VOP time 
base increment encoder, step 44, where the remainder of the 
presentation time code of the P-VOP is encoded. 

The process then repeats with the next encoded video 
object plane which is die B-VOP. The time code comparator, 
step 42, compares die presentation time of the B-VOP 
against the B-VOP time base register. The result is passed to 
die modulo time base encoder, step 43. The modulo time 
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base encoder will insert the required number of "1" equiva- 
lent to the number of modulo time base increment that has 
elapsed. This is then followed by the symbol "O" to signal 
the end of the modulo time base code. Both the B-VOP time 
base register and the local time base register are not changed 5 
after the processing of the B-VOP. The processing then 
proceed to the VOP time base mcrement encoder, step 44, 
where the remainder of the presentation time code of the 
B-VOP is encoded. 

The local time base register is reset at the next I-VOP lO 
which marks the beginning of the next group of VOP. 
Implementadon of the Bitstream Decoder is described 
FIG. 12 illustrates the block diagram for the implemen- 
tation of the decoder for the modulo time base and VOP time 
increment to recover the presentation time stamp. As in the 15 
implementation of the encoder the example in FIG. 3B is 
used. The decoding order is the same as the encoding order 
where the I-VOP is decoded followed by the P-VOP before 
the B-VOP. This is explained in the paragraphs below. 

The processing begins in the initializer, step 51, where the .20 
local time base register is set to the value of tiie time code 
decoded from the bitsu-eam. The processing then proceeds 
the modulo time base decoder, step 52, where the modulo 
time base increment is decoded. The total number of modulo 
time base increment decoded is given by the number of "1" 25 
decoded before the symbol **0" Next the VOP time base . 
increment is decoded in the VOP time base increment, step 
53. In the time base calculator, step 54, the presentation time 
of the I-VOP is recovered. The total decoded modulo time , 
base increment value is added to die local time base register. 30 
The VOP time base increment is then added to the local time 
base register to obtain the presentation time of the I-VOP. 
The process then goes to the video object decoder where the 
video object is decoded. 

For the P-VOP, the process repeats in the modulo time 35 
base decoder, step 52, where the modulo time base incre- * 
raent is decoded. The total number of modulo time base ■ 
increment decoded is given by the number of "1" decoded ; 
before the symbol "O". Next the VOP time base increment , 
is decoded in the VOP time base increment, step 53. In the 40 
time base calculator, step 54, the presentation time of the • 
P-VOP is recovered. The B-VOP modulo time base register 
is set to the value in die local- tis?:i':base register. The total 
decoded modulo time base increment value is added to the 
local time base register. The VOP time base increment is 45 
then added to the local time base register to obtain the . 
presentation time of the P-VOP. The process then goes to the 
video object decoder where the video object is decoded. 

For the B-VOP, the process repeats m the modulo time 
base decoder, step 52, where the modulo time base incre- 50 
ment is decoded. The total nimiber of modulo time base 
increment decoded is given by the number of "1" decoded 
before the symbol "O". Next the VOP time base increment 
is decoded in die VOP time base increment, step 53. In the 
time base calculator, step 54, the presentation time of the 55 
B-VOP is recovered. The total decoded modido time base 
increment value and the VOP time base increment is added 
to the B-VOP time base register to obtain the presentation 
time of die B-VOR Both die B-VOP time base register and 
die local time base register are left unchanged The process 60 
then goes to the video object decoder where die video object 
is decoded. 

The local time base register is reset at die next I-VOP 
which marks die beginning of die next group of VOP. 

Specific example is described. 65 
Referring to FIG. 13, an example of the steps for encoding 
the compressed data into bitstream data is shown. As shown 



in the top row in FIG. 13, die compressed video data VOPs 
are aligned in die display order, U, Bl, B2, PI. B3, P3 widi 
a GOP (group of pictures) header inserted at the beginning 
of group of VOPs. While being displayed, die local time, at 
which the display is effected, is measured for each VOP 
using die local time clock. For example, die first VOP 
(Il-VOP) is displayed at 1 hour 23 minute 45 second 350 
millisecond (1:23:45:350) counted from die very beginning 
of die video data; die second VOP (Bl-VOP) is displayed at 
1:23:45:750; die diird VOP (B2-V0P) is displayed at 
1:23:46:150; and so on. 

For encoding die VOPs, it is necessary to insert the 
display time data to each VOP. The insertion of die full tune 
data including hour, minute, second and millisecond occu- 
pies a certain data space in the header portion of each VOP. 
The aim of die present invention is to reduce such a data 
space and to simplify the time data to be inserted in each 
VOP. 

Each of the VOPs shown in die first row in FIG. 13 is 
stored with the display time data of millisecond at the VOP 
time increment area. Each of die VOPs in die first row is also 
stored temporarily with die display time data of hour, minute 
and second The GOP header is stored widi die display data 
of hour, minute and second for die first VOP (Il-VOP). 

As shown in the second row in FIG. 13, die VOPs are 
delayed by a predetermined time using a buffer (not shown). 
When die VOPs are produced firom die buffer, die order of 
the VOPs are changed according to die bi-directional pre- 
diction rule such diat die bi-directional VOP, i.e., B-VOP, 
should be positioned after die P-VOP to which die B-VOP 
references. Thus, the VOPs are aligned in die order of II, PI, 
Bl, B2. P2, B3. 

As shown in die diird row in FIG. 13. at time Tl, i.e., 
when the GOP header is being encoded, die local time base 
register stores the hour, minute, second data as stored in the 
GOP header. In the example shown in FIG. 13, the local time 
base register stores 1:23:45. Then, before time T2, die 
bitstream data for die GOP header is obtained widi die hour, 
minute, second data being inserted as shown at the bottom 
row in FIG. 13. 

Then, at time T2. die first VOP ai-VOP) is taken up. The 
time code comparator compares the time (hour, minute, 
second) stored in die local time base register widi the time 
(hour, minute, second) temporarily stored in the first VOP 
(Il-VOP). According to die example, die result of die 
comparison is die same. Thus, die comparator produces **0" 
indicating diat die first VOP (Il-VOP) occurs in die same 
second as the second being held in the local time base 
register. The result **0" as produced fix>m the comparator is 
added to die first VOP (Il-VOP) at die modulo time base 
area. At the same time, die hour, minute and second data 
temporarily stored in die first VOP Ol-VOP) is eliminated. 
Thus, before time T3. die bitstream data for die first VOP 
(Il-VOP) is obtained with **0" inserted in the modulo time 
base area and "350" inserted in die VOP time increment 
area. 

Then, at time T3, die second VOP (Pl-VOP) is taken up. 
The time code comparator compares the lime (hour, minute, 
second) stored in die local time base register widi die time 
(hour, minute, second) temporarily stored in die second VOP 
(Pl-VOP). According to the example, die result of die 
comparison is such diat die time temporarily stored in die 
second VOP (Pl-VOP) is one second greater dian die time 
stored in local time base register. Thus, die comparator 
produces "10" indicating diat die second VOP (Pl-VOP) 
occurs in the next one second to die second being held in die 
local time base register. If die second VOP (Pl-VOP) occurs 
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in the next— next second to the second being held in the After time T6. the B-VOP time base register is incre- 
local time base register, the comparator will produce **110". mented to the time equal to the time carried in the local time 
After time T3, the B-VOP time base register is set to the base register immediately before time T6. In the example, 
time equal to the time carried in the local time base register the B-VOP time base register is incremented to 1:23:46. 
immediately before time T3. In the example, the B-VOP 5 Also, after time T6, the local time base register is incre- 
time base register is set to 1:23:45. Also, after time T3, the mentfcd to the time equal to the time temporarily stored in the 
local time base register is incremented to the time equal to f^fth vOP (P2-V0P). Thus, in tiie example, die local time 
the time temporarily stored in the second VOP (Pl-VOP). ^^^^ register is incremented to 1:23:47. 
Thus, in the example, the local time base register is incre- ^he result "10" as produced from tiie comparator is added 
mented to 1:23:46. ... 10 to the fifth VOP (P2-V0P) at the modulo time base area. At 

The result "10** as produced from the comparator is added ^^^^ ^^^^ 3^^^^^ ^j^ta temporarily 

to the second VOP (Pl-VOP) at the modulo time base area. ^^^^^ y^p (p2.vOP) is eliminated Thus, before 

At the same time the h^^; ^^J^^^^^^ w^^T.^ time T4, the bitstream data for the fifth VOP (P2-V0P) is 
i^y stored m the second VOP (Pf-y.^P) ^^^^^^ obtained witii "10" inserted in the modulo time base area and 

Thus, before tune T4, die bitstream data for me second VOP „^ -^i* • j T/nT> ;«/*r.*«,o«t 
(Pl-VOP) is obtained with "10" inserted in the modulo time ^5 "350' mserted m the VOP time mcrement area 
base ^ea ^d "550" inserted in die VOP time increment TTiereafter, a smu^ar operation is ca^^ out for formmg 

bitstream data for the foUowmg VOPs. 
Then at time T4, the third VOP (Bl-VOP) is taken up. For decoding the bitstream data, an operation opposite to 
The time code comparator compares die time (hour, minute, die above is carried out. First, die time (hour, mmute, 
second) stored in die B-VOP time base register widi die time 20 . second) carried in die GOP header is read. The read time is 
(hour, minute, second) temporarily stored in die durd VOP stored in die local time base register. 
(Bl-VOP) According to die example, die result of die Upon receipt of die 1 type or P type VOP, Le., odier dien 
comparison is die same. Thus, die comparator produces "0" die B type VOP. die data stored in die modulo time base area 
indicating diat die third VOP (Bl-VOP) occurs in die same is read. If die read data is "0". i.e., widiout any is in front of 
second as die second being held in die B-VOP time base 25 0. no change is made in die local time base register, and also 
register. The result "0" as produced from die comparator is , no change is made in die B-VOP time base register. If die 
added to die diird VOP (Bl-VOP) at die modulo time base ' read data is "10*', die time stored in die local time base 
area. At die same time, die hour, minute and second data register is incremented by one second. If die read data is 
temporarily stored in die diird VOP (Il-VOP) is eliminated. /UIO", die time stored in die local time base register is 
Thus, before time T5. die bitstream data for die diird VOP 30 ; incremented by two seconds. In diis manner, die number of 
(Bl-VOP) is obtained widi "0" inserted in die modulo time seconds which should be incremented is determined by die 
base area and "750" inserted in die VOP time increment number of is inserted in front of 0. Also, when die read data 

is "10" or "110", die B-VOP time base register, which is a 
Then, at time T5, die fourth VOP (B2-V0P) is taken up, . memory, copies die time die local time base register had 
The time code comparator compares die time (hour, minute, 35 carried just before die most recent increment Then, die time 
second) stored in die B-VOP time base register widi die time • (hour, minute, second) carried in die local time base register 
(hour, minute, second) temporarily stored in die fourth VOP is combined widi die time (millisecond) carried in die VOP 
(B2-V0P). According to die example, die result of die • time increment area to establish a specific time at which die 
comparison is such diat die time temporarily stored in die I type or P type VOP should occur, 
fourdi VOP (B2-V0P) is one second greater dian die time 40* Upon receipt of die B type VOP. die data stored in die 
stored in B-VOP time base register. Thus, die comparator • modulo time base area is read. If die read data is "0**, die 
produces " 10" indicating diat die fourth VOP (B2-V0P) time (hour, minute, second) carried in die B-VOP VOP time 
occurs in die next one second to the second being held in die . base register is combined widi die time (milliseo5^.?-d) carried 
B-VOP time base register. in t^e VOP time increment area to establish a specific time 

During die processing of die B type VOP. neidier die local 45 at which die B type VOP should occur. If die read data is 
time base register nor die B-VOP time base register is . "10". die time (hour, minute, second) carried in die B-VOP 
incremented regardless of whatever result die comparator time base register is added one second, and dien die added 
produces. result time is combined with die time (millisecond) carried 

The result "10" as produced from die comparator is added in die VOP time increment area to establish a specific time 
to die fourth VOP (B2-V0P) at the modulo time base area. 50 at which die B type VOP should occur. If die read data is 
At die same time, die hour, minute and second data tempo- "110", die time (hour, minute, second) carried in die B-VOP 
rarily stored in die fourdi VOP (B2-V0P) is eliminated time base register is added two seconds, and dien die added 
Thus, before time T6. die bitstream data for die fourth VOP result time is combined widi die time (millisecond) carried 
(B2-V0P) is obtained widi "10" inserted in die modulo time in die VOP time increment area to estabUsh a specific time 
base area and "150" inserted in die VOP time increment 55 at which die B type VOP should occur. 

The effects of this invention is to enable video object 
Then, at time T6, die fifdi VOP (P2- VOP) is taken up. The planes encoded by different encoders to be multiplexed. It 
time code comparator compares die time (hour, minute. will also facilitate object based bitstream manipulation of 
second) stored in die local time base register widi die time compressed data diat come from different sources to create 
Oiour, minute, second) temporarily stored in die fifdi VOP 60 new bitstrcams. It provides a mediod of synchronization for 
(P2-V0P). According to the example, die result of die audio visual objects. 

comparison is such diat die time temporarily stored in die The mvention being dius described, it will be obvious diat 
fifdi VOP (P2-V0P) is one second greater dian die time die same may be varied in many ways. Such variations are 
stored in local time base register. Thus, die comparator not to be regarded as a deparmre from die spirit and scope 
produces "10" indicating diat die fifdi VOP (P2-V0P) 65 of die invention, and all such modifications as would be 
occurs in die next one second to die second being held in the obvious to one skilled in die art are intended to be included 
local time base register. widiin die scope of die following claims. 



